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Tlorough comparative case study, we seek to understand the ways 
in which actors in high schools use and think about performance 
data. In particular, we compare data use in higher and lower 
value-added schools. Data use is conceptualized here as having 
access to a host of available performance data on students, using 
them to guide instructional decisions, and building cultures in 
which data are seen as vehicles for improvements in practice. While 
our findings do not show consistent differences in the higher and 
loiver value-added schools, they do allow us to set out an expanded 
conceptualization of school data use. 


INTRODUCTION 

Federal and state policies that reward or sanction schools for improving 
test scores present educators with significant incentives for using student 
performance data. In this environment, school leaders are using data to make 
instructional and resource decisions. Faculty are encouraged to “teach to the 
test” by aligning curricula and materials to the content and format of state 
tests (Au, 2007, 2008; Lyons & Algozzine, 2006). School leaders schedule to 
the test, reducing time allocated for lunch, recess, and some untested subjects 
Qennings & Stark, 2006; Rentner et al., 2006). They also “staff to the test,” 
moving to tested grades teachers whose students make substantive learning 
gains (Cohen-Vogel, 2011). Despite the growing prevalence of data use in 
educational organizations, however, relatively little is known about whether 
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and how the process of integrating data into practice differs between more 
and less effective schools. 

The data for this article are drawn from a larger study of high school 
effectiveness by the National Center on Scaling up Effective Schools; this 
larger study focuses on eight components of school effectiveness rang¬ 
ing from quality instruction to personalized learning connections (Goldring, 
Porter, Murphy, Elliott, & Cravens, 2009). Here, we focus on findings related 
to one such component—data use in schools. 

Through comparative case study, we seek to understand the ways in 
which educators in high schools use and think about performance data as 
they make decisions in increasingly complex schooling contexts. In partic¬ 
ular, we are interested in comparing data use by schools that are growing 
students’ performance with those that are not. We conducted this work in 
Florida, where accountability is relatively high stakes and performance data 
have been collected for over a dozen years. Specifically, we were guided by 
the following questions: What kinds of data are available to educators and 
how do they access these data? In what ways do teachers and administrators 
use data, and how do leaders work to build the capacity of their staffs to 
leverage data for instructional improvement? And, are there systematic differ¬ 
ences in the ways educators in higher value-added and lower value-added 
schools access, use, and understand performance data? 

To answer these questions, we begin by synthesizing the existing 
research to construct a comprehensive framework for understanding data 
use in schools. Our review suggests a trifold framework that includes the 
kinds of information that educators integrate into their practice, the ways in 
which they use that information, and the cultures they build to support data- 
driven practice. We describe the framework in the following section. We then 
discuss the data collection and analytic methods employed in our case study 
work and introduce the four schools in which data were collected. Finally, 
we present our findings. 


FRAMING DATA USE IN EFFECTIVE HIGH SCHOOLS 

Today’s educators operate in information-rich environments in which numer¬ 
ous performance data exist that may inform decision-making and improve 
efforts to bolster student achievement (Anderson, Leithwood, & Strauss, 
2010). In order to better understand the role of information in schooling, 
we construct a framework for systematic data use that emphasizes three 
key elements. First, educators in effective high schools have access to and 
use a wide variety of performance data; further, they are afforded easy 
access to such data through comprehensive information management sys¬ 
tems. Second, personnel across organizational levels in effective schools use 
data for a diverse set of purposes; effective schools also develop the capacity 
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of their staffs to do so through focused professional development. Finally, 
the use of performance data in effective schools is increasingly mediated by 
strong, positive cultures of data use, in which educators work together to 
use available information for school improvement and collectively construct 
positive perceptions of the benefit and utility of data use in their practice. 
Each of these elements, when in place, supports the development of schools 
as effective and efficient “learning organizations.” 

Data Access and Availability 

Research supports the idea that a wide variety of performance data are 
available to school educators (Firestone & Gonzalez, 2007; Guskey 2003; 
Halverson, Grigg, Prichett, & Thomas, 2007; Ingram, Louis, & Schroeder, 
2004; Louis, Leithwood, Wahlstrom, & Anderson, 2010). Further, these data 
are derived from multiple sources. Teachers, school leaders, and support per¬ 
sonnel may, for instance, have access to data from external sources, like state 
or district performance assessments. Firestone and Gonzalez (2007) note that 
these data sources are “central to modern accountability initiatives”; indeed, 
the focal nature of such data sources within accountability structures may 
lend them substantial authority (p. 133). Additionally, as Anderson et al. 
(2010) note, such externally generated data may carry additional weight due 
to their perceived objectivity, validity, and reliability within the “academic 
test and measurement community” (p. 295). 

Research (Black & Wiliam, 1998; Firestone & Gonzalez, 2007) also 
indicates that internally generated performance data are pivotal to educa¬ 
tors’ efforts to engage in diagnostic assessment of student learning, and 
in improving educational outcomes. Firestone and Gonzalez (2007) iden¬ 
tify both formal and informal manifestations of such data. They note, for 
instance, that informal sources like “teachers’ observational or anecdotal 
records” offer information that “can help adjust the delivery of instruction”; 
similarly, they argue, more formal data sources like “class projects, home¬ 
work assignments, and end-of-unit tests all provide information about what 
students have learned” (p. 134). Black and Wiliam (1998) underscore the 
importance of internally generated, often informal, sources of data, asserting 
that it is “important to look at or listen carefully to the talk, the writing, 
and the actions through which pupils develop and display the state of 
their understanding . . . for this will initiate the interaction through which 
formative assessment aids learning” (p. 7). 

Individual stakeholders may not, however, value internal and exter¬ 
nal performance data equally. Some research (e.g., Guskey, 2007; Ingram 
et al., 2004) indicates, for example, that teachers may question the valid¬ 
ity of standardized assessments, constructing an environment in which 
“being dismissive of externally generated achievement data is a cultural trait 
that teachers learn and pass on to other teachers” (Ingram et al., 2004, 
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p. 1273). Other studies (e.g., Anderson et al., 2010; Cohen-Vogel, 2011; 
Cohen-Vogel, Osborne-Lampkin, & Houck, 2013; Cohen-Vogel & Rutledge, 
2009; Guskey, 2007) suggest that district and school administrators are 
more open to utilizing externally derived data, primarily for evaluative and 
staffing purposes. Interviewing principals across a number of school con¬ 
texts, Louis et al. (2010) finds that the majority report relying on “state- and 
district-mandated” measures of student achievement to inform a number of 
decisions—including those related to professional development. 

In terms of data access, the literature also indicates that educators are 
accessing these diverse performance data through the use of increasingly 
complex information management systems (Cohen, 2003; Louis et al., 2010; 
Gallagher, Means, & Padilla, 2008). By and large, however, these systems 
are not uniform in their comprehensiveness, and may be limited in terms of 
the types of data they offer to practitioners (Gallagher et al., 2008; Means, 
Padilla, DeBarger, & Bakia, 2009). Cohen (2003) reports that recent efforts to 
expand access to state-level educational data are encouraging, and cites the 
Education Data Warehouse (EDW) movement as a prime example of how 
easy access to data may motivate educators to make wider use of available 
information. Petrides and Guiney (2002) support this, noting that the devel¬ 
opment of such one-stop-shops for data have ameliorated the issues caused 
by less efficient data systems, in which “departments and offices in schools 
maintain independent sources of data with these sources rarely related to 
each other” leading to “data redundancy and inaccuracies” (p. 7). 

The Capacity for Data Use and Action 

A number of researchers argue that developing capacity for data use among 
teachers is vital in establishing effective data-driven practice (e.g., Halverson 
et al., 2007; Kerr, Marsh, Ikemoto, Darilek, & Barney, 2006; Louis et al., 2010; 
Murnane, Sharkey, & Boudett, 2005; Wohlstetter, Datnow, & Park, 2008). 
Gallagher et al. (2008), for example, find that teachers who report high lev¬ 
els of confidence in their ability to use data and greater school support for 
professional development focused on data use are significantly more likely to 
report using performance data to communicate with parents, track student 
performance, identify skill gaps, and control instructional pacing (p. 20). 
Louis et al. (2010) find that, in schools exhibiting greater propensity for data 
use, “principals and teachers report increasing efforts to develop the capacity 
of teachers to engage collectively in data analysis for instructional decision 
making”; lower data use schools, conversely, report a strong reliance on 
external “experts” (p. 192). Further research (Gallagher et al., 2008; Means 
et al., 2009; Murnane et al., 2005; Sharkey & Murnane, 2003) suggests, 
however, that even as districts seem to be gradually improving support 
structures for data use, including professional development, many educa¬ 
tors, particularly teachers in schools not meeting AYP, still feel unprepared 
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and undertrained to use data to engage in activities like interpreting test 
scores, adjusting the curriculum, developing diagnostic assessments, and 
interrogating data in meaningful ways. 

Studies also show that educators translate their capacity to use data into 
practice in a variety of ways (Anagnostopoulos & Rutledge, 2007; Firestone 
& Gonzalez, 2007; Lyons & Algozzine, 2006). Firestone and Gonzalez (2007) 
construct a broad typology of data uses, asserting that within local orga¬ 
nizations data serve to guide instructional actions, enlighten teachers and 
school leaders, and mobilize support for decisions. Examples of the first 
categorization, “guidance for action,” have recently been documented in 
the literature. Cohen-Vogel (2011), for instance, finds that school leaders in 
Florida report “using students’ scores on the state standardized exam ... to 
make teacher staffing decisions”; she reports that “professional development, 
too, is planned with student assessment data in hand” (p. 499). Gallagher 
et al. (2008) find that teachers report using data to monitor student progress 
and communicate that progress to parents; they also find, however, that rel¬ 
atively fewer teachers report using data to identify skill gaps in students, 
change their curriculum, or modify their practices. Louis et al. (2010) sup¬ 
port this, finding that the majority of principals and teachers in their study 
use data for “problem identification” rather than “problem solving.” 

Few studies have, to date, shown linkages between data-driven practices 
and improved student achievement. Anderson et al. (2010) suggest, however, 
that current conceptions of data-based practice in schools may be construed 
too narrowly and that positive effects may be more likely as teachers, admin¬ 
istrators and other school personnel begin to go “beyond the identification 
of problem areas to an investigation of the specific nature of and factors 
contributing to [problems]” (p. 321). Black and Wiliam’s (1998) review of 
research finds that, as teachers develop their capacity to use “formative,” 
internally-generated data to refine and improve their practice, they often pro¬ 
duce “significant and often substantial learning gains” (p. 3). Further, they 
assert that teachers’ use of formative data to improve their practice “helps 
low achievers more than other students,” closing achievement gaps while 
improving overall achievement (Black & Wiliam, 1998, p. 3). They describe 
the state of such data use in schools as mired in a “poverty of practice,” 
however, and assert that many educators require significant development in 
their ability to leverage “formative” data in their work (p. 4). 

Cultures of Data Use 

Researchers on the vanguard of the field are pushing beyond simply dis¬ 
cussing the development of systems and capacity for data-driven practice 
in schools, and are beginning to explore what we call here the culture of 
data use in educational organizations (Firestone & Gonzalez, 2007; Ingram 
et al., 2004; Sutherland, 2004; Wohlstetter et al., 2008). As schools evolve into 
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“learning organizations,” administrators may cultivate a culture of data use 
in which teachers are trained to “see the value of data” and school leaders 
construct “explicit norms and expectations regarding data use” that “foster 
mutual accountability” (Wohlstetter et al., 2008, p. 247, 255). As Sutherland 
(2004) states, under such a positive culture of data use, making reflective 
and systematic decisions based on information becomes “the way we do 
things around here” and data changes from “something that is done to the 
school” to “something that is done by the school and for the school” (p. 289). 
Ultimately, such positive cultures of data use promote an atmosphere of 
“organizational learning,” in which emphasis is placed on “improved instruc¬ 
tion, problem solving, and an investment in the long term that incorporates 
teachers’ and principals’ voices” (Firestone & Gonzalez, 2007, p. 152). 

Researchers also describe positive cultures of data use as highly collabo¬ 
rative in nature (Huffman & Kalnin, 2003; Petrides & Guiney, 2002; Sharkey & 
Murnane, 2003). Petrides and Guiney (2002), for instance, assert that collab¬ 
orative data cultures embrace “communities of practice” in which educators 
construct knowledge ecologies that “weave together the actions of building 
a vision, stating the school’s mission, and engaging in reflective practice and 
inquiry” (p. 1710). Wohlstetter et al. (2008) find that in districts with posi¬ 
tive cultures of data use, teachers “rely heavily on one another for support, 
new instructional strategies, and discussions about data”; they also assert that 
“structured time around data discussions was probably the most important 
scaffolding for continuous improvement” in the study schools (p. 253). 

We conceptualize systematic data use, therefore, as a confluence of 
three parts: having access to a host of available data on students, using 
data to guide instructional actions and mobilize support for decisions, and 
building cultures in which data are seen as vehicles for improvements in 
practice. Using this framework, we conducted interviews and focus groups 
to understand data use in both higher and lower value-added schools. More 
specifically, we wanted to understand whether the schools participating in 
our study were operating in the kinds of diverse knowledge ecologies iden¬ 
tified by existing research. We also sought to identify the ways in which 
various actors in our case study schools, both higher and lower value-added, 
utilized data in their practice and how their capacity to do so was devel¬ 
oped. Finally, we sought to understand the ways in which participants in 
our case study schools worked together to build cultures of data use, and 
how those cultures mediated their use of data in improving student learning 
and performance. 


METHODS 


Case Study Schools 

Working in Florida, we began by using a simple value-added achievement 
model (YAM) to estimate the relative performance of all of the state’s high 
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schools. Here, we were interested in comparing the practices of schools 
that were making gains with minority students, low-income students, and 
English Language Learners (ELL), in particular. The estimated fixed effect for 
each high school in the state was put in rank order and classified by deciles 
of value-added for these groups (see Sass, 2012). These analyses identified 
only one Florida district with multiple high- and low-value-added schools, 
according to our definition of school effectiveness. We partnered with this 
district for our work. The district has been engaged in a high school reform 
effort for the past nine years and achieved national recognition for its efforts 
to improve chronically low-performing schools. Despite these successes, 
the district had repeatedly failed to meet overall reading proficiency goals, 
as well as reading and mathematics proficiency goals for African American, 
economically disadvantaged and ELL-eligible students. 

Four high schools in the district—two higher value-added and two 
lower value-added—were selected for case study using a ranked list from 
the VAM analysis (see Sass, 2012). From the list, we eliminated alternative, 
charter, and magnet high schools and then selected two higher and lower 
value-added schools for case study. With a strategy that balanced perfor¬ 
mance with demographic characteristics, we selected the two lowest ranked 
schools on math and reading gains with low-income, minority, and ELL stu¬ 
dents. We then selected the two highest ranked schools that had comparable 
student demographics. 

Overall, the schools were relatively similar to one other on tradi¬ 
tional demographic dimensions. They were all large (Florida has the largest 
schools, on average, in the nation), and had majority-minority student pop¬ 
ulations and average percentages of ELL for the district. School 102, one of 
the lower value-added schools, was slightly smaller than the others, with 
somewhat larger proportions of students qualifying for free or reduced-price 
lunch (FRPL) and ELL services. 

The two lower value-added high schools in our study were 101 and 102. 
School 101 enrolled approximately 2,100-2300 students. In 2010, approxi¬ 
mately 45-55% of those students qualified for FRPL; additionally, between 
55-65% of the student body were identified as minorities. Five to ten per¬ 
cent of the school’s students were classified as ELL in 2010. School 101’s 
school grade, as defined by Florida’s accountability framework, vacillated 
between an “A” and a “B” over the last three academic years. 1 During the 
2010-2011 academic year, the school was classified as being in corrective 
status under the state’s accountability system. 

During the 2010-2011 school year, the second of our lower value-added 
schools, 102, had approximately 1800-2000 students. Sixty to seventy percent 
of those students qualified as FRPL. Sixty to sixty-five percent of the student 
body at 102 was of minority descent. Between ten and fifteen percent of the 
school’s students were classified as ELL. School 102’s school grade shifted 
between a “D” and a “C” over the last several years, and it was placed in 
corrective status by the state’s DA system in 2010. 
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TABLE 1 Demographic Profile of Case Study Schools, 2010-2011. 


School 

Enrollment 

% FRPL 

% 

% ELL 

101 

2100-2300 

45-55 

55-65 

5-10 

102 

1800-2000 

60-70 

55-65 

10-15 

103 

2200-2400 

45-55 

65-75 

5-10 

104 

2800-3000 

30-40 

50-60 

5-10 

District average 

2327 

52 

73 

9 


Note. Schools 103 and 104 were identified as higher value-added using our value-added model; schools 
101 and 102 are lower value-added. To ensure confidentiality, school names are pseudonyms, and 
demographic values have been converted to ranges. Demographic data were derived from the School 
Public Accountability Reports (SPAR) compiled by the Florida Department of Education. FRPL = Free 
and Reduced Price Lunch; Minority African American, Hispanic, Asian, & Other; ELL = English Language 
Learners. District averages were calculated with data from all non-alternative, non-charter public high 
schools in the district (A = 29). 


The first of our higher value-added case study schools, 103, 2 had 
approximately 2200-2400 students in 2010 (see Table 1). Between 45-55% 
of those students qualified for FRPL. The student body was predominantly 
comprised of students of minority descent, who accounted for 65-75% of the 
school’s enrollment. School 103’s school grade was an “A” for the last sev¬ 
eral years; additionally, the schools’ relatively high performance resulted in 
its assignment to a category in the state’s differentiated accountability (DA) 
system with less oversight and monitoring than other schools in the study. 
One element that differentiated 103 from the other schools in the study was 
its system of admission—students matriculated to 103 through a “lottery” sys¬ 
tem. Like many large, urban districts, the sampled district offered a variety 
of choice options from magnets to charters to schools within schools. While 
there were no performance criteria on which applicants were evaluated at 
103, and the school was required to maintain a demographic makeup con¬ 
sistent with the wider district, there was an element of “choice” to 103’s 
admissions process not present at the other sampled schools. 

Finally, the second of our higher value-added schools, 104, served 
between 2800-3000 students in 2010. Thirty to forty percent of those stu¬ 
dents qualified for FRPL. Students of minority descent comprised 50-60% of 
the student body. Between five and ten percent of the school’s students were 
classified as ELL. School 104’s school grade was a “B” over the last three aca¬ 
demic years and the school was in corrective status in 2010, as determined 
by the state of Florida’s DA system. 


Data 

Data analyzed in this study were collected from our four case study schools 
during three week-long visits in the 2010-2011 school year. In total, these 
data represented 120 participant interactions across the four schools, and 
provided a broad perspective of the perceptions and reported practices of 
educators, across organizational levels of each school, related to data analysis 
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TABLE 2 Data Sources by School. 


Interaction Type 

School 101 

School 102 

School 103 

School 104 

Interviews 

Principals 

2 

2 

2 

2 

Assistant principals 

4 

5 

4 

4 

Guidance counselors 

2 

2 

2 

2 

Support personnel 

2 

2 

2 

2 

Department heads 

4 

4 

4 

3 

Teachers 

9 

9 

9 

9 

Students 

3 

3 

3 

3 

Focus groups 

Teachers 

2 

2 

2 

2 

Students 

2 

2 

2 

2 

Total 

30 

31 

30 

29 


and decision making. Specifically, we drew on data from 104 semi-structured 
interviews and 8 focus groups with teachers or students (see Table 2). 
We worked with principals and guidance counselors on question develop¬ 
ment and piloted the protocols with teachers and administrators in a second 
district before data collection began. 

In each school, interviews were conducted with the principal, assistant 
principals, guidance counselors, support personnel (e.g. lead instructional 
coordinators and ELL coordinators), students, and department heads and 
teachers of mathematics, science, and English language arts. With regard 
to data use, we asked participants about the roles that data play in their 
practice, and included items for teachers, for example, such as “How do you 
use data in your classroom?” 

Our work was iterative. Throughout the data collection process, we 
interrogated the data in order to identify emerging concepts and avenues for 
further inquiry. This work was facilitated through the use of Post-Interaction 
Forms (PIFs), which prompted us to consider the ways in which partici¬ 
pant’s responses supported or refuted the essential components framework 
and to identify emerging concepts or issues with the data collection process 
(see discussion of post-held reflection activities in Miles & Huberman, 1994). 
Further, on a twice-weekly basis during the fieldwork, the three members 
of an individual school teams met and used their PIFs to prepare a school 
report synthesizing their collective work. After each full held visit, school 
teams took these heldnotes and worked together across schools to compile 
preliminary hndings, revise interview or focus group protocols, and design 
further data collection activities. 

Analysis 

Following this iterative process of data collection and preliminary analysis, 
we began systematically analyzing data through directed content analysis 
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(Patton, 2002). We started by analyzing the data categorically, first assigning 
basic, descriptive codes for data availability and access , data capacity and 
use, and/or culture of data use, to reflect the three component parts of our 
conceptual model. Within each component category, we allowed themes to 
emerge from the data inductively (Miles & Huberman, 1994). In so doing, we 
cataloged the various data types as internal and external and informal and 
formal, and the uses of data for staffing decisions, instructional adaptation, 
and needs identification, among others. Basic codes identifying cultures of 
data use were similarly expanded, to include secondary codes like data use 
as a schoolwide norm and collaborative analysis and use. 

In addition to this descriptive coding, we also engaged in summative 
content analysis of the interview and focus group data (Hsieh & Shannon, 
2005). As part of this analytic process, rubrics were constructed which 
researchers used to assign numeric scores for the intensity or frequency 
with which certain elements of systematic data use were reported by our 
participants. The rubric elements evaluated by coders were developed col- 
laboratively by the analysts and included: the diversity of demonstrated use 
(based upon the number of discrete uses for data identified by the respon¬ 
dent), the frequency of demonstrated use, perceptions of the culture of data 
analysis and use (rating the positivity or negativity of the respondents’ 
expressed perception of the use of data in their school), and the frequency 
of collaborative data analysis and use. Mean scores across these rubric cate¬ 
gories were computed and compared using independent samples T-tests to 
compare higher- and lower-value-added schools. 

Two people coded the data to promote reliability in the coding process. 
During the initial phase of analysis, the coding pair analyzed and scored the 
same transcripts on the basic, descriptive codes, noting possible codes for 
the second level of analysis. Additionally, coders sought out and identified 
discontinuing evidence—what Corbin and Strauss (2008) term “cases that 
don’t fit the pattern” (p. 298). The coding pair met frequently to share and 
test these emergent codes, work through coding inconsistencies, refine the 
coding framework and rubrics, and build reliability. Once inter-rater relia¬ 
bility was established, analysts coded the data. During this second phase, 
the coding team continued to meet to discuss potential issues in the coding 
process, additional emergent codes, and any disconfirming evidence. In the 
case of negative or disconfirming evidence, the team worked collaboratively 
to revise their coding framework—either modifying construct definitions 
or eliminating constructs when appropriate. Additionally, the coding pair 
wrote two types of memos during the coding process—an in-depth anno¬ 
tated memo, citing coded evidence and rubric scores to describe findings 
at each school and noting negative cases in the data, and summary memos 
that identified similarities and differences across cases and the policies, pro¬ 
grams, and practices related to data use at each school. As we describe in 
the findings section, these procedures led to the extension of our guiding 
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framework by Firestone and Gonzalez (2007), and, specifically, allowed us 
to identify two manifestations of schools use of data to guide action. 


SYSTEMATIC USE OF DATA: FINDINGS FROM FOUR CASE-STUDY 

SCHOOLS 

For the most part, we found more commonalities than differences between 
data use by the higher and lower value-added schools. There were some 
differences, however, between three schools and the fourth, whose perfor¬ 
mance put it in an advanced improvement status, subjecting it to state and 
district oversight and regular visits by the district area supervisor and state 
institutional response teams. We present findings for the three components 
of our framework each in turn. 


DATA AVAILABILITY AND ACCESS 

Participants across all four case-study schools reported that they are sur¬ 
rounded by performance data from numerous sources. Administrators, teach¬ 
ers, and support personnel alike indicated that they have ready access to 
externally derived data like AP scores, scores on the Florida Comprehensive 
Assessment Test (FCAT), and scores on district benchmark tests. A depart¬ 
ment head at 101, for example, highlighted the role of these data in her 
practice, sharing that “I think we are primarily driven by FCAT data for every¬ 
thing in the school. FCAT data, and of course . . . we have to look at pass 
rates [on the AP exam] for each course.” Internally derived information—like 
student grades, attendance data, teachers’ formative perceptions of student 
performance, and information from casual observations and conversation— 
also reportedly played a significant role in educators’ work across all four 
institutions. School 103’s principal, for example, shared that he always finds 
time to talk with students to gauge the health of the school, asserting that 

I can find out what’s going on in the classroom just by walking around 
in the cafeteria ... I will have a kid come up to me in the cafeteria and 
I say “Why aren’t you in class?” “What class are you in?” “Um, chemistry.” 

“Your teacher let you out? There is like 35 minutes left in the period!” 
“Yeah, she said I could leave ... I was done.” That tells me that the 
teacher doesn't teach bell to bell. 

Participants in all four schools reported that they accessed these data 
through two primary gateways to the district’s data “warehouse.” The first, 
Virtual Counselor, is an online data system offering principals and teach¬ 
ers access to a wide variety of information, including students’ attendance 
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records, academic history, schedules, and test scores. The second, Pinnacle, 
offers online access for students and parents to information on students’ 
progress on course assignments and grades. One teacher in 103 detailed the 
comprehensive nature of Virtual Counselor, sharing that the system is 

a resource website, where it shows the credits the student has taken, the 
track they are on, their grades in every class. You can look up FCAT data 
on Virtual Counselor. You can look at their schedule. You can look up 
every test they have had. It’s like a counselor on computer. All of the 
records on that child for everything. 

Typical of comments across the schools, a student shared that Pinnacle is 
her parent’s primary school monitoring tool: 

My mom, she set up this thing with Pinnacle that it like updates her every 
five minutes, and she always goes on. If the teacher puts in an early grade 
[for a student], like before the rest of us turn it in, [my mom] will say, 

“Why didn’t you hand it in?” I am like, “We didn’t hand it in yet!” She 
stays on top of things to keep me on top of things. 

Participants in 104 reported that their access to both internally and exter¬ 
nally derived data is enhanced through a third system, in addition to Virtual 
Counselor and Pinnacle, developed within the school itself. The system 
incorporates scores on a series of formative assessments—derived from the 
district’s benchmark tests—into the wider pool of data provided by other 
systems. 

In general, the relative value placed on these types of data was also 
fairly consistent across schools. Educators, on the whole, adopted a bal¬ 
anced approach to selecting from available information in their day-to-day 
decisions. Consistent with prior research, however, teachers in all four 
schools expressed a greater reliance on, and trust in, internally generated 
performance data, like teacher-developed assessments, students’ grades, and 
informal feedback from students, than did their administrators (e.g., Guskey, 
2007; Ingram et al., 2004). A teacher in 101 summarized this perspective, 
asserting that, while externally derived data “helps to understand,” “that’s 
not who those kids are. They are living, breathing things. They are people. 
They have feelings and emotions.” This perceptual divide was reportedly 
much more pronounced in 102, with a number of participants asserting that 
the views of the faculty differed sharply from the administration, particularly 
the principal, regarding the relative emphasis that should be placed on dif¬ 
fering types of data. One teacher in the school expressed frustration with the 
principal’s focus on externally derived data, asserting that “all he talks about 
is numbers. The kids to him mean numbers. I guess for him all he wants is 
to keep his job and make sure . . . we meet AYP, so he looks good. I think 
that’s all he wants, honestly.” 
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On the whole, we found few systematic differences regarding the ways 
participants in our four case study schools access data, or the nature of 
the data that is available to them. In both higher and lower value-added 
schools, participants reported that they are immersed in information-rich 
environments, and that the data available to them includes externally derived 
data like standardized test scores and internally derived data like classroom 
assessments. They also reported accessing data through the same informa¬ 
tion management systems. While many teachers reported holding a greater 
trust in internally derived data—especially in 102, one of the lower value- 
added schools—school actors as a whole seem to have adopted a balanced 
approach to selecting from available information. 


Data Use and Capacity for Action 

Participants across all four case-study schools reported that their use of 
data largely falls within what Firestone and Gonzalez (2007) termed “guid¬ 
ance for action.” Based on the data gathered in our case study schools, we 
extended the Firestone and Gonzalez framework here by categorizing guid¬ 
ance for action into two subconstructs—wherein schools used data to guide 
the structure of the learning environment, and to inform the instruction of 
students. 

In all four schools, participants said that performance data—particularly, 
externally derived data—play a key role in structuring the learning environ¬ 
ment by informing decisions regarding student placement. This is largely 
enabled by a district-created “assignment matrix” that uses student course 
grades and scores on the FCAT to assign students to courses—AP, Honors, 
Regular, and Remedial. An assistant principal in 103 described the process 
this way: 

Each student is assigned [to courses] based on their performance. 

We believe it’s important to have an appropriate match of curriculum to 
the student’s ability. So students who have the ability to take Advanced 
Placement classes will be enrolled in Advanced Placement classes or 
Honors classes and so forth. If a student shows a deficiency, whether it 
be in math or reading or writing, then they will be enrolled in a class 
to help conquer that deficiency and increase their achievement. So all 
assignments are based on the student data. 

Also related to the structure of the learning environment, participants’ 
reports in three of the four schools—101, 102, and 103—suggested that data 
are used to assign teachers to grade levels and tracks. An assistant principal 
in 101, for example, shared that the process of assigning teachers to courses 
revolves around questions like: ‘“Based on the data, tell us, do you think 
teacher X is okay to handle the Algebra classes next year?’ ‘Let’s look at his 
or her learning gains for the students, and see if that’s a good fit or not.’ 
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Most of our teachers are placed into classes accordingly.” The principal of 
102 offered a similar example: 

For the last three years, I have looked at my writing teachers’ scores on 
the FCAT, and I have some teachers that are just like consistently in the 
70, 80 percent of 3.5s and 4s. Then over the last few years, I have had 
teachers that were not. So I made some really conscious decisions. I took 
a couple of people out of 10th grade writing, meaning the 10th grade 
English teachers, and I put some other people in. I had one person that 
only had one section [of 10th grade English), but she had great scores. 

So the next year she had all of 10th grade [a tested grade]. 


Participants across all four schools also reported that data guide their 
instructional efforts. Instructional efforts are expressed in two dominant 
forms: the targeting of certain students for instructional intervention and the 
modification of classroom practice. By and large, participants were much 
more likely to report that data—both externally derived assessment data and 
formal, internally derived data like grades or benchmark tests—are used to 
target groups of students for intervention or additional services than to report 
using them for instructional modification. Numerous participants in all four 
schools, for example, reported that they use data to identify students in the 
“bottom 30%” for “pull-out” courses or Saturday FCAT Camps. 

A department head in 103 offered another example of “targeting” 
students for instructional intervention: 

After spring break, we are going to have our “crunch time tutoring” for 
our level-three students. Students that have just made FCAT level three. 

Our concern is that sometimes these kids fall back because they don't 
have a reading class, so those students—it’s about 170 of them, 9th and 
10th graders—they are going to get pulled from their elective class start¬ 
ing when we come back from spring break, and I will be tutoring them 
up to the test. 


On the whole, participants in all four schools reported that they use 
performance data—both externally and internally generated—to modify 
classroom instruction far less frequently than they use it to “target” or identify 
student groups. Participants in 103, one of the higher value-added schools, 
reported one of the few cases of systemic instructional adaptation based 
on data. Members of the administration there indicated that scores from 
the district’s benchmark assessments are used to tailor schoolwide “do-now” 
activities aimed at specific deficiencies in student performance. Several teach¬ 
ers, across all four case-study schools, indicated that they used internally 
generated, often informal, data to guide instructional decisions—generally 
regarding instructional pacing or targeting areas of deficiency. A teacher in 
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102, for example, shared that “from a writing prompt, on the first day of 
school, I can tell already [there’s] no need [for] working with subject-verb 
agreement. These kids got it. What do they need? ... I base it on what I 
see from the kids.” Participants did not, however, indicate that there was a 
systematic effort to incorporate such data into decision making across their 
schools, framing the practice as a largely individual effort. 

In addition to the purposes for which data are being used, researchers 
were interested in the ways schools are working to build capacity for data 
use. Participants in three of the four schools—101, 103, and 104—reported 
that there are systematic supports in place to build educators’ capacity to 
make use of performance data in their practice. Participants at 102, on the 
other hand, reported that such efforts to develop the capacity of all school 
actors to use data are highly infrequent and not systemic. 

In 103, efforts to build data-use capacity are enacted through “data 
chats” at the beginning of the year, in which an assistant principal, counselor, 
and teachers from across academic departments meet to collectively analyze 
data. A department head at the school described the activity as follows: 

We were with the reading department, and, I believe, the foreign 
language department. We were all in one room with two guidance coun¬ 
selors and we went through our data. We had to fill out these sheets that 
basically told us which strands or benchmarks the kids were lowest in. 

What were your lowest, medium, and highest [strands], so that we could 
formulate a plan the very first day of school. 

Participants in 101 reported that they engage in a similar process. The 
school’s principal shared that 

We have chats with the teachers during their planning periods. We talk 
about three things every three weeks. [At] our most recent one, we talked 
about identifying students that are in your lowest quartile when you look 
at the data. And, subgroups—identifying who are those kids, where are 
we with them, and what’s your approach in teaching them. And, looking 
through the lesson planning piece. Our next chat focuses on what are 
you doing with those kids, give me specific best practices that you are 
doing to share among the staff. 

In 104, participants reported that capacity development centers on students’ 
performance and is a responsibility of the recently formed instructional lead¬ 
ership team. The instructional leadership team uses the data to identify 
teachers with whom they may work to help identify student deficiency areas 
and model instruction. 

Participants at 102, by and large, did not report that there are systematic 
efforts to develop the capacity of the school staff to use data to inform their 
practice. One assistant principal indicated that he is trying to build up his 
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teachers’ ability to use data, and that they “work a lot on data and being 
able to break down what’s going on.” He also indicated, however, that the 
opportunities available to do so are “not enough” and not schoolwide. 

Overall, our findings did not indicate that systematic differences exist 
in the ways educators in the higher and lower value-added schools in the 
study use data to guide and inform their practice. In all four of our case-study 
schools, participants indicated that data play a variety of roles in their work, 
providing valuable information used to structure the learning environment 
and, to a lesser extent, instruct their students. Analysis of the rubric scores 
supported findings from the qualitative coding described throughout this 
section (see Table 3). In comparing mean rubric scores for participants’ use 
of data, we found no significant differences between higher and lower value- 
added schools for either what participants use data for (diversity of use in 
Table 3) or how often they use them (frequency of use). Moreover, in both 
of our higher value-added and one of our lower value-added schools, study 
participants reported that there are systematic efforts to build the capacity of 
the instructional staff to use data in their practice. One school did stand out, 
however. According to participants at 102, there are few efforts to develop 
the faculty’s capacity for data use. 


Cultures of Data Use 

The extent to which our case-study schools have developed strong and 
positive cultures of data use seems to vary across schools. Participants in 
103 reported the “strongest” culture, indicating that they tend to view data 
as a beneficial part of their practice. One teacher in the school, for exam¬ 
ple, enthusiastically reported that “It’s everything. Data drives everything . . . 
That’s the basis for everything I do.” 

Participants in 101 and 104 expressed somewhat mixed perceptions of 
the benefit and utility of data use, and indicated that data cultures are still 
growing and evolving in their organizations. Several teachers in 101, for 


TABLE 3 Mean Scores on Rubric Dimensions by School. 


Dimension 

School 

101 

School 

102 

School 

103 

School 

104 

HVA/LVA 

Difference 

Data use and capacity for action 

Frequency of use 

2.19 

2.10 

2.23 

2.44 

No 

Diversity of use 

2.11 

2.10 

2.14 

1.96 

No 

Cultures of data use 

Perception 

2.05 

1.48 

2.42 

2.09 

Yes 

Frequency of collaboration 

2.18 

1.85 

1.67 

2.13 

No 


Note. Schools 103 and 104 were identified as “higher value added” using our value-added model. The 
HVA/LVA Difference column indicates whether mean differences between higher value-added schools 
and lower value-added schools are significant at the p < .05 level. 
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example, reported that they are required to use data, and generally view it 
as necessary; some also indicated, however, that they do not hold a positive 
perception of how data is being used in the school: 

We have to look at the kids who are low performing. How do we take 
this particular group of kids, and how do we move them to the next level? 

What do we do with it? I don't think we use data effectively. I think for 
the most part we collect it. We look at it. We talk about it and that’s it. 

Educators in 104 reported similar perceptions. One teacher, for instance, said 
“I am going to be frank here. The jumping through hoops to meet certain 
requirements of testing, and meetings about your lowest quartile and your 
AYP—I mean, sometimes I feel like if they would leave us alone and let us 
teach we would be okay.” Another was more positive, asserting that: “When 
I see my kids, I could tell you off the top of my head what they are weak in 
. . . [for example,] informational text. So, I think that has been a tremendous, 
positive change. Knowing where your students are, where their weaknesses 
are, where their strengths are.” 

102’s culture of data use stands in sharp contrast to the cultures in our 
other case-study schools. Teachers in the school expressed generally nega¬ 
tive perceptions regarding data-driven practice. The negative culture seems 
to be driven by a strong emphasis on accountability and a practice of “sham¬ 
ing” teachers with data. Teachers reported feeling extremely anxious about 
the results of external performance measures, sharing that “When I get the 
scores still, it takes my breath away. It makes me feel very worried, anxious 
about what’s going on.” Others expressed frustration. They perceive that the 
accountability system is based on one-sided or invalid data. According to 
one teacher, for example, “They don’t even look at the data. Not your data. 
The data that they collect ... all this home-grown data, they haven’t looked 
at that.” Another teacher agreed: 

Every week or so there is pre- and post-assessments, and then there is 
training and another round of testing. They want me to look at this data 
and plan my instruction around it and I can’t. The data isn’t valid. I know 
what my kids’ issues are. I see it in the work they turn in. That’s where 
my instruction needs to take place. 

At 102, underperforming teachers are also reportedly “named and 
shamed.” Here, students’ scores on the state assessment are linked with 
teachers’ names and shared publicly with other teachers in the school. 
According to one teacher in the school: “My data is published at the end 
of the year. It’s made public. I mean that in a loose sense. Not to the whole 
community, but other teachers can see what kind of learning gains did [I] get 
last year.” Another teacher reported that data 
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end up around the whole school. I am not in [another teacher’s] depart¬ 
ment, but I know what percentage he has, and I know his score. It was 
pointed out: “Should we do what we normally do and hide the names?” 

[The principal replied,] “No, leave it like that. I want everybody to see 
the names.” That was a decision that was made, and I can only think it 
was to embarrass people into doing better. 

In considering data-use cultures, researchers were also interested in col¬ 
laborations among teachers and other staff members in schools. In all four 
sampled schools, participants reported that there are structures in place to 
encourage collaborative analysis and use of performance data. These largely 
center on “data chats” between administrators and teachers. Small Learning 
Communities and Professional Learning Communities were also named as 
venues in which teachers use data collaboratively. The frequency with which 
actors engage in collaborative data use, and the benefit they perceive in 
doing so, varies between schools, however. Participants in 101, for instance, 
reported that teachers and administrators collaboratively engage with data on 
a frequent basis—about every three weeks. Some participants in the school, 
however, reported that they feel these activities are less than useful. One, for 
instance, said that 

We have data chats, which can get a little frustrating . . . Sometimes you 
sit in a meeting for an hour. It gets around to your turn so you talk for 
five minutes about what you did in your class that day, or whatever, and 
in the mean time, to be quite honest, in your mind you are like, “I could 
have graded a whole stack of papers.” 

Participants in 103 and 104 reported that they engage in collaborative data 
use on a more infrequent basis—a few times per year in 103, for instance— 
but that these activities tend to be productive and focused on instructional 
practice. Reports at 102 stood in relative contrast to the other schools. While 
“data chats” were cited as an avenue for collaboration around data, partici¬ 
pants said they occur infrequently and serve to bolster an already negative 
climate. 

As with the two other elements of the framework, there do not appear 
to be consistent and systematic differences between our higher and lower 
value-added schools in terms of data cultures. In three of the four schools, 
participants indicated that they operate in functional cultures of data use. 
While participants in 103 reported a more generally positive perception of 
the benefit and utility of data in their practice, those in 104 and 101 seemed 
to merely accept data-driven practice as the new norm. Similarly, participants 
in all four case-study schools indicated that there are structures in place that 
serve to support the collaborative analysis and use of available performance 
data. Once again, however, 102 stands out from the other case-study schools. 
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Relative to the other three schools, participants there reported that the culture 
of data use in which they operate is significantly more negative and focused 
on shaming, and that they often question the benefit, utility, and even validity 
of data-driven practice in their school. Comparisons of the mean rubric scores 
for participants’ perception of the benefit and utility of data use does indicate 
that there is a significant difference between the higher and lower value- 
added schools in our study (see Table 4). But, as Table 4 shows, and the 
responses reported above reflect, this appears to be driven by higher scores 
in 103 and lower scores in 102. 


DISCUSSION AND CONCLUSION 

Without exception, participants in all four schools reported that they had easy 
access to a comprehensive pool of performance data. They also reported 
using data in similar ways, ways that we grouped into two categories: to 
guide the structure of the learning environment and to guide instruction. 
According to participants, all four schools also employed some structures by 
which data were analyzed and used collaboratively through data chats, for 
example, although the nature of these collaborations appear to have differed 
somewhat between schools (see Table 4 for a summary of findings). 

There did not appear to be clear and consistent differences in the ways 
the three elements of our data-use framework played out in the higher and 
lower value-added schools. There were, however, differences that distin¬ 
guished one lower value-added school from the others. Interpreting this 
finding, we suggest that three of the schools in our sample, including the 
two schools identified as higher value added and another identified as lower 
value-added, seemed to embed their efforts to make data accessible and 
build capacity for its use into a broader culture of “organizational learning,” 
as described by Firestone and Gonzalez (2007). Leaders in these schools 
appeared to use positive data cultures to promote an atmosphere of learn¬ 
ing that emphasized continuous improvement and a long-term vision shared 
by teachers and principals. Our fourth case study school, 102, identified 
through our value-added model as lower value-added, seemed to be mired 
in a very different context of data use, however; study participants there 
reported that they often found themselves laboring under the negative effects 
of an “audit” or “accountability-focused” data culture. Here, data was con¬ 
structed as being the province of the administration, used to monitor and 
“embarrass” faculty and hold them accountable. This resulted in a negative, 
almost caustic, environment, with participants expressing a lack of confi¬ 
dence in performance data, the utility of data collaboration, and the ability 
of the principal to promote the school’s success. 

We cannot draw causal conclusions from case-study designs like this 
one. However, because we did not see consistent differences between the 


TABLE 4 Summary Findings by Key Framework Dimensions and School. 
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’. Schools 103 and 104 were identified as higher value added. Differences among schools are italicized. 
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higher and lower value-added schools, the study suggests that whether a 
school has had success in improving the achievement scores of low-income, 
minority, and ELL students may not be related to the types of data available 
to schools, the ways those data are used, and the data cultures within them. 
If these conclusions are confirmed in future studies, in other contexts and 
with other designs, the implication is that policies aimed at building data 
systems to be used in schools along with teachers’ capacity to use them will 
not be enough to leverage broad-scale improvement. 

The findings from our four case-study schools extend our knowledge 
regarding the ways in which schools engage in data-driven practice and 
promote functional cultures of data use in four ways. First, the study expands 
on the work of Firestone and Gonzalez (2007). In particular, it teases apart 
what these authors refer to as using data as guidance for action into two key 
types, specifying the ways in which educators leverage data to inform the 
structure of the school’s learning environment, on one hand, and to influence 
instructional practice, on the other. 

Second, by looking at the ways data are used to guide the structure 
of the learning environment and instructional practices separately, our study 
documented additional uses as well as a proclivity among schools toward 
some uses over others. In terms of using data to structure the learning envi¬ 
ronment, schools were employing performance data in the assignment of 
students to courses and teachers to grade levels and tracks. While under¬ 
standing whether changes to the structure of the learning environment were 
“good” or “bad” for students was outside the scope of the study in terms of 
the data available to us, future projects might consider the extent to which 
the availability of student-level performance data and schools’ proclivity for 
its use legitimizes and perhaps even expands various forms of ability group¬ 
ing. On one hand, using performance data to place students into courses 
might circumvent the subjectivity (and biases) of teacher recommendations. 
On the other, it may work to solidify tracking systems in high schools and 
encourage departmentalization in lower grades. Cohen-Vogel (2011) found 
that three of ten elementary schools studied were experimenting with depart¬ 
mentalized structures, wherein students in grades as low at 2nd rotated 
to different teachers for math, science, and reading. In two of these three 
schools, departmentalization was coupled with tracking. According to the 
author, “at an F school, where student math scores were contributing to a 
low school grade, the principal had just implemented class-by-class tracking 
for mathematics instruction in third grade as part of a new supplementary 
mathematics curriculum” (p. 494). 

In their use of data to guide instructional practice, study participants 
in all four schools were much more likely to report using both internally 
and externally derived data to guide decisions about which students to tar¬ 
get for educational interventions and services (e.g., pull-out help, extended 
learning opportunities, and tutoring) than for modifying their own classroom 
instruction. While many teachers, in particular, did report that they used 
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informal, often formative data to shape their practice, these efforts were 
described as being individual and non-systematic in nature. This suggests that 
while data and the performance-based accountability movement in general 
may have seeped more deeply into “the core technology of schooling” than 
institutionalists and others who study schools as organizations (e.g., Ogawa, 
2009) may have predicted, there appears to be what we here might term “lim¬ 
ited data saturation” into schools. That is to say, that even as data—especially, 
in this case, the externally derived, formal types of data favored under most 
state accountability frameworks—are being used in and by schools to guide 
decisions about what is taught and by whom it is taught, they have not as 
of yet motivated teachers to change their instructional practices. It may also 
be that teachers are not modifying instruction because they do not know 
how to do so; the evidence for which instructional practices work best with 
different types of learners is still relatively limited. 

Third, our study expands upon previous research on data cultures 
(e.g., Petrides & Guiney, 2002; Wohlstetter et al., 2008) by describing actors’ 
perceptions regarding the benefit of data-driven practices, as well as efforts 
by schools to foster collaborative data use among members of their faculty. 

Finally, we show that consideration of any one of the elements alone— 
data availability and access, data use and capacity for action, and data 
cultures—may not be sufficient to understand the patterns of data use 
in today’s public schools. Instead, the three elements of our framework 
appear to interact and combine to shape schools into varying data-use 
environments—“organizations of learning,” on one hand, focused on the 
use of data to empower new ideas about teaching and learning, or “audit 
cultures,” on the other, focused on the use of data for evaluation and 
accountability. Future studies of data use should apply our expanded frame¬ 
work and look more closely at the factors that condition the development of 
these environments in schools. 


NOTES 


1. The state’s grading system incorporates various performance indicators, only a small portion 
of which measure value added. As such, it is possible that our lower value-added schools—as measured 
by value added to the achievement of three traditionally underperforming subpopulations—may receive 
high or average grades by the state. 

2. Schools were given pseudonyms to protect confidentiality. 
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